Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1645 Improve Spark Streaming compatibility with Flume
  3. SPARK-1729

Make Flume pull data from source, rather than the current push model

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.1.0
    • DStreams
    • None

    Description

      This makes sure that the if the Spark executor running the receiver goes down, the new receiver on a new node can still get data from Flume.

      This is not possible in the current model, as Flume is configured to push data to a executor/worker and if that worker is down, Flume cant push data.

      Attachments

        Activity

          People

            hshreedharan Hari Shreedharan
            tdas Tathagata Das
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: